{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "465fc80f",
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import openpifpaf\n",
    "openpifpaf.show.Canvas.show = True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3f44cfb",
   "metadata": {},
   "source": [
    "# Animal Keypoints\n",
    "\n",
    "<div style=\"text-align: right\"> by <a href=\"https://scholar.google.com/citations?user=f-4YHeMAAAAJ&hl=en\">Lorenzo Bertoni</a> , 12/05/2021 </div> <br /> \n",
    "\n",
    "This section describes the [OpenPifPaf](https://github.com/openpifpaf/openpifpaf) plugin for animals. The plugin uses the [Animalpose Dataset](https://sites.google.com/view/animal-pose/) that includes 5 categories of animals: dogs, cats, sheeps, horses and cows. For more information, we suggest to check our latest [paper](https://arxiv.org/abs/2103.02440): <br /> \n",
    "\n",
    "> __OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association__<br />\n",
    "> _[Sven Kreiss](https://www.svenkreiss.com), [Lorenzo Bertoni](https://scholar.google.com/citations?user=f-4YHeMAAAAJ&hl=en), [Alexandre Alahi](https://scholar.google.com/citations?user=UIhXQ64AAAAJ&hl=en)_, 2021.\n",
    ">\n",
    "\n",
    "\n",
    "## Setup\n",
    "```sh\n",
    "pip3 install openpifpaf\n",
    "```\n",
    "(in case using CUDA 9 specify: `pip install torch==1.7.1+cu92 torchvision==0.8.2+cu92 -f https://download.pytorch.org/whl/torch_stable.html`)\n",
    "\n",
    "## Predict\n",
    "Prediction runs as standard openpifpaf predict command; the only difference is the pretrained model.\n",
    "To download the model, just add `--checkpoint shufflenetv2k30-animalpose` to the prediction command.\n",
    "\n",
    "For example on a picture of our researcher Tappo Altotto:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db2bb87c",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "python -m openpifpaf.predict images/tappo_loomo.jpg --image-output \\\n",
    "    --checkpoint=shufflenetv2k30-animalpose \\\n",
    "    --line-width=6 --font-size=6 --white-overlay=0.3 --long-edge=500"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5c84594",
   "metadata": {},
   "outputs": [],
   "source": [
    "import IPython\n",
    "IPython.display.Image('images/tappo_loomo.jpg.predictions.jpeg')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0fec712",
   "metadata": {},
   "source": [
    "The standard skeleton of an animal is the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32588751",
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [],
   "source": [
    "# HIDE CODE\n",
    "# first make an annotation\n",
    "ann_animal = openpifpaf.Annotation.from_cif_meta(\n",
    "    openpifpaf.plugins.animalpose.animal_kp.AnimalKp().head_metas[0])\n",
    "\n",
    "# visualize the annotation\n",
    "openpifpaf.show.KeypointPainter.show_joint_scales = False\n",
    "openpifpaf.show.KeypointPainter.line_width = 3\n",
    "keypoint_painter = openpifpaf.show.KeypointPainter()\n",
    "with openpifpaf.show.Canvas.annotation(ann_animal) as ax:\n",
    "    keypoint_painter.annotation(ax, ann_animal)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "274bc55e",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Preprocess dataset\n",
    "The Animalpose dataset is in a mixed format. Some annotations comes from a collection of keypoints annotations of [PASCAL Dataset](http://host.robots.ox.ac.uk/pascal/VOC/) from UC Berkeley, which the authors of the animalpose dataset have extended. Other annotations have been created by the authors of the [dataset](https://sites.google.com/view/animal-pose/). As no datasets split was available, we created a random training/val split with 4000 images for training, and the remaining 600+ for validation.\n",
    "\n",
    "The following instructions will help with downloading and preprocessing the Animalpose dataset.\n",
    "\n",
    "```sh\n",
    "mkdir data-animalpose\n",
    "cd data-animalpose\n",
    "pip install gdown\n",
    "gdown https://drive.google.com/uc\\?id\\=1UkZB-kHg4Eijcb2yRWVN94LuMJtAfjEI\n",
    "gdown https://drive.google.com/uc\\?id\\=1zjYczxVd2i8bl6QAqUlasS5LoZcQQu8b\n",
    "gdown https://drive.google.com/uc\\?id\\=1MDgiGMHEUY0s6w3h9uP9Ovl7KGQEDKhJ\n",
    "wget http://host.robots.ox.ac.uk/pascal/VOC/voc2011/VOCtrainval_25-May-2011.tar\n",
    "tar -xvf keypoint_image_part2.tar.gz\n",
    "tar -xvf keypoint_anno_part2.tar.gz\n",
    "tar -xvf keypoint_anno_part1.tar.gz\n",
    "tar -xvf VOCtrainval_25-May-2011.tar\n",
    "rm keypoint_*.tar.gz\n",
    "rm VOCtrainval_25-May-2011.tar\n",
    "cd ..\n",
    "```\n",
    "\n",
    "After downloading the dataset, run:\n",
    "```sh\n",
    "python3 -m openpifpaf.plugins.animalpose.scripts.voc_to_coco\n",
    "```\n",
    "\n",
    "## Training\n",
    "```sh\n",
    "pip install pycocotools\n",
    "pip install scipy\n",
    "```\n",
    "\n",
    "```sh\n",
    "python3 -m openpifpaf.train \\\n",
    "  --lr=0.0003 --momentum=0.95 --clip-grad-value=10.0 --b-scale=10.0 \\\n",
    "  --batch-size=16 --loader-workers=12 \\\n",
    "  --epochs=400 --lr-decay 360 380 --lr-decay-epochs=10 --val-interval 5 \\\n",
    "  --checkpoint=shufflenetv2k30 --lr-warm-up-start-epoch=250 \\\n",
    "  --dataset=animal --animal-square-edge=385 --animal-upsample=2 \\\n",
    "  --animal-bmin=2 --animal-extended-scale \\\n",
    "  --weight-decay=1e-5\n",
    "```\n",
    "\n",
    "## Evaluation\n",
    "To evaluate our model using the standard COCO metric, we use the following command:\n",
    "```sh\n",
    "python3 -m openpifpaf.eval \\\n",
    "    --dataset=animal --checkpoint=shufflenetv2k30-animalpose \\\n",
    "    --batch-size=1 --skip-existing --loader-workers=8 --force-complete-pose \\\n",
    "    --seed-threshold=0.01 --instance-threshold=0.01\n",
    "```\n",
    "\n",
    "## Everything else\n",
    "All PifPaf options and commands still stand, check them in the other sections of the guide.\n",
    "If you are interested in training your own dataset, read the section on a {doc}`custom dataset <plugins_custom>`.\n"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "ea6946363a43e80d241452ab397f4c58bdd3d2517da174158e9c46ce6717422a"
  },
  "kernelspec": {
   "display_name": "Python 3.9.4 64-bit ('venv3': venv)",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": ""
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
